iT邦幫忙

2022 iThome 鐵人賽
DAY 25
4
DevOps

淺談DevOps與Observability系列 第 25 篇

OTel Collector + Loki + Grafana

14th鐵人賽 opentelemetry grafana fluentbit loki
2471 瀏覽

  • 分享至 

  • xImage
    •  

今天來搭配Docker來跑看看!
還沒使用OTel collector之前, 大概大家都這樣做

我用mingrammer/flog這是apache log產生器
kscarlett/nginx-log-generator這個則是nginx log產生器

程式結構

/
  docker-compose.yaml
  conf/
      fluent-bit.conf
      parsers.conf

version: '3'
services:
  fluentbit:
    image: fluent/fluent-bit
    volumes:
        - ./conf:/fluent-bit/etc
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    hostname: fluentd

  flog:
    image: mingrammer/flog
    command: '-l'
    depends_on:
      - fluentbit
    command: "-s 2 -n 5"
    logging:
      driver: fluentd
      options:
        tag: apache
  nginx:
    image: kscarlett/nginx-log-generator
    depends_on:
      - fluentbit
    environment:
      - RATE=1
      - DELETE_PERCENT=5
    logging:
      driver: fluentd
      options:
        tag: nginx

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false

fluent-bit.conf

[SERVICE]
    flush 1
    Daemon off
    log_level debug
    http_server on
    http_listen 0.0.0.0
    storage.metrics on
    Config_Watch On
    parsers_file parsers.conf


[INPUT]
    Name forward
    Listen 0.0.0.0
    port 24224
    
[FILTER]
    Name parser
    Match apache
    Key_Name log
    Parser apache
[FILTER]
    Name record_modifier
    Match apache
    Record log_type apache

[FILTER]
    Name parser
    Match nginx
    Key_Name log
    Parser nginx

[FILTER]
    Name record_modifier
    Match nginx
    Record log_type nginx

[OUTPUT]    
    name                   loki    
    match                  *    
    host                   loki   
    port                   3100    
    labels                 job=demo

[OUTPUT]
    Name stdout
    Format json

parsers.conf

[PARSER]
    Name   apache
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

然後執行起來

docker-compose up -d

瀏覽器打開http://localhost:3000/login
預設帳號密碼admin/admin

左邊選擇DataSources, 然後新增, 選擇Loki
晚點會把它寫成provisioning!!
URL填入http://loki:3100,
按下Save&Test

接著到Explore, DataSource選擇Loki
就能看到剛剛在FluentBit新增的job demo了

想練習前三天介紹的LogQL, 這裡就有很多資料能把玩了:)

加入OTel Collector!

這樣子的架構是因為, FluetBit有提供OpenTelemetry的Output組件.
也有提供Loki的Output組件
真心強大的FluentBit:)

但我們也能這樣玩

直接把程式的log, 輸出到OTel collector上, 再匯出到Loki.
OTel Collector有提供Fluent Forward Receiver
(但這條路目前崎嶇難行就是QQ)

程式結構

/
  docker-compose.yaml
  otel/
    otel-collector-config.yml

docker-compose.yaml

version: '3'
services:

  flog:
    image: mingrammer/flog
    command: '-l'
    depends_on:
      - otel-collector
    command: ["--delay=2s", "--number=1", "--format=json", "--loop"]
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        # Allow time for otel-collector to spin up, then forward fluentd logs to the fluentforwarder receiver.
        fluentd-async-connect: "true"
        # Use nanosecond precision
        fluentd-sub-second-precision: "true"
        tag: apache
  nginx:
    image: kscarlett/nginx-log-generator
    depends_on:
      - otel-collector
    environment:
      - RATE=1
      - DELETE_PERCENT=5
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        # Allow time for otel-collector to spin up, then forward fluentd logs to the fluentforwarder receiver.
        fluentd-async-connect: "true"
        # Use nanosecond precision
        fluentd-sub-second-precision: "true"
        tag: nginx

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    
  otel-collector:
    image: otel/opentelemetry-collector-contrib
    command: ["--config=/etc/otel-collector-config.yml"]
    volumes:
      - ./otel/otel-collector-config.yml:/etc/otel-collector-config.yml
    ports:
      - "1888:1888"   # pprof extension
      - "13133:13133" # health_check extension
      - "55679:55679" # zpages extension
      - "24224:24224" # fluentforwarder
      - "24224:24224/udp" # fluentforwarder
    depends_on:
      - loki

otel-collector-config.yml

receivers:
  fluentforward:
    endpoint: 0.0.0.0:24224

processors:
  batch:
    send_batch_size: 50
    timeout: 5s
  memory_limiter:
    check_interval: 2s
    limit_mib: 1800
    spike_limit_mib: 500


extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679
  memory_ballast:
    # Memory Ballast size should be max 1/3 to 1/2 of memory.
    size_mib: 64

exporters:
  logging:
      loglevel: debug
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"
    # Will encode the whole OTEL message in json
    # This format happen after extracting labels
    format: json
    labels:
      attributes:
        container_name: ""
        source: ""
      resource:
        host.name: "hostname"

service:
  extensions: [pprof, zpages, health_check, fluentbit]
  pipelines:
    logs:
      receivers: [fluentforward]
      processors: [ memory_limiter, batch ]
      exporters: [loki, logging]

然後執行起來

docker-compose up -d

瀏覽器打開http://localhost:3000/login
預設帳號密碼admin/admin

左邊選擇DataSources, 然後新增, 選擇Loki
晚點會把它寫成provisioning!!
URL填入http://loki:3100,
按下Save&Test

接著到Explore, DataSource選擇Loki
就能看到剛剛在FluentBit新增的label了

查詢了一下能發現到body的問題

嘗試用前天提到的json parsing expression

問題依舊

能善用也是前幾天提到的line_format, 只提取出body形成新的log後再json parse

{container_name="/demo_log_flog_1"} | json  | line_format "{{.body}}" | json

可是FluentBit不是也能寫一堆parser嘛? 參考Log Agent - Fluent Bit Parser元件
怎麼還要在這用LogQL轉?

因為這裡
Deprecated FluentBit Subprocess Extension
FluentBit的OTel collector extension目前被棄用了QQ

所以之前的那個架構還是有存在的必要, 先到FluentBit弄好後再往Collector拋. 但就是更複雜了.

最後一步, Grafana Provisioning for DataSources

DataSource確定能連線後, 我們應該就要建立成Provisioning, 方便進入版控, 拉下來佈署時就能直接配置好了(CAC).

程式結構

/
  docker-compose.yaml
  otel/
    otel-collector-config.yml
  provisioning/
    datasources/
      loki.yaml

loki.yaml

# config file version
apiVersion: 1

datasources:
  - name: Loki
    type: loki
    url: http://loki:3100

docker-compose.yaml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - ./provisioning/datasources/:/etc/grafana/provisioning/datasources/
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false

一登入成功後, Loki就在data sources內了:)

Provisioning能回憶我前幾天的文章
淺談Provisioning Grafana

今日小心得

hmm...FluentBit真心好用（喂）
OTelCellector真的支持很都多種協議跟接口, 不過在選用之前還是得要注意是否有狀態是Deprecated的.
該狀態也能回顧這次小弟的文章淺談OpenTelemetry

但真的不失為是一個高度整合性的解決方案.

參考資料

OpenTelemetry Collector Contrib

2021年小弟的FluetBit系列


上一篇
Loki LogQL - Metric Queries
下一篇
Grafana - 為Log與Trace搭起一座鵲橋
系列文
淺談DevOps與Observability 共 36 篇
  1. 32
    淺談Grafana Tempo DataFlow
  2. 33
    一起閱讀Grafana Tempo - Configuration Section 1
  3. 34
    一起閱讀Grafana Tempo - Configuration Section 2
  4. 35
    一起閱讀Grafana Tempo - Configuration Section 3
  5. 36
    The OpenTelemetry Bootcamp
完整目錄
圖片
  直播研討會
圖片
{{ item.channelVendor }} {{ item.webinarstarted }} |
{{ formatDate(item.duration) }}
直播中

1 則留言

0
sunny70487
iT邦新手 5 級 ‧ 2023-01-31 11:03:41

您好，想請問在opentelemetry-contrib下的traceid、spanid可以直接在log內生成hashcode嗎？還是需要藉由trace工具才能達成呢？謝謝。

雷N iT邦研究生 1 級 ‧ 2023-01-31 14:05:10 檢舉

你好, 如果是要透過traceid+spanid兩個組成hashcode
可以的, 能參考小弟這篇https://ithelp.ithome.com.tw/articles/10288979 的baggage or span tag
把hashcode塞進去即可

若是要拿traceid or span id能看看span context
https://opentelemetry.io/docs/reference/specification/trace/api/#retrieving-the-traceid-and-spanid

登入發表回應

我要留言

立即登入留言
iThome鐵人賽
參賽組數
1064 組
團體組數
40 組
累計文章數
22195 篇
完賽人數
600 人
iT+看影片追技術 看影片追技術 看更多
熱門tag 看更多
15th鐵人賽 16th鐵人賽 13th鐵人賽 14th鐵人賽 12th鐵人賽 11th鐵人賽 鐵人賽 2019鐵人賽 javascript 2018鐵人賽 python 2017鐵人賽 windows php c# windows server linux css react vue.js
熱門問題
熱門回答
熱門文章
IT邦幫忙